Skip to content

feat: register Fable 5.2, GPT-6.2 Sol and GPT-6.2 Astra models - #46

Merged
samuelboland merged 2 commits into
mainfrom
dux-newmodels-w1-01-fable-5-2-gpt-6-2-registry
Nov 18, 2026
Merged

samuelboland merged 2 commits into
mainfrom
dux-newmodels-w1-01-fable-5-2-gpt-6-2-registry

Conversation

@staging-supernova-dx-appf-io

@staging-supernova-dx-appf-io staging-supernova-dx-appf-io Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

What this does

Adds three newly available models to the fork's catalog (PR #46, NewModels W1-01). main is the 1.16 line that Supernova runs today.

  • Claude Fable 5.2 (anthropic and bedrock)
  • GPT-6.2 Sol and GPT-6.2 Astra (OpenAI on Bedrock-mantle)

No lib/ code changed. Routing already matches the new ids. Providers::Bedrock::MANTLE_ONLY_MODEL_PATTERN, Protocols::MantleResponses::FRONTIER_GPT_PATTERN and Protocols::Responses::Chat::GPT6_MODEL_PATTERN already match openai.gpt-6.2-sol and openai.gpt-6.2-astra. Fable 5.2 uses the same Converse and InvokeModel paths as Fable 5.1. The registry entries, aliases and specs encode live-probe results from 2026-11-18 (account 436292035443).

Commit 5a39210: registry and offline specs

  • lib/ruby_llm/models.json gains five entries: claude-fable-5-2 (anthropic), global.anthropic.claude-fable-5-2 and us.anthropic.claude-fable-5-2 (bedrock), and openai.gpt-6.2-sol and openai.gpt-6.2-astra (bedrock).
    • They were added as text by an insertion script that checks no existing entry changed. It printed inserted 5 entries; 1243 -> 1248.
    • models.json was not reformatted or regenerated.
    • The reviewer re-ran the insertion check against aa95f613. Five entries were added and no existing entry changed.
  • lib/ruby_llm/aliases.json gains claude-fable-5-2 (anthropic and bedrock), gpt-6.2-sol and gpt-6.2-astra.
  • Fable 5.2 prices per million tokens (input / output / cache_read / cache_write):
    • Global and anthropic ids: 8 / 40 / 0.20 / 10.
    • us. id: 8.8 / 44 / 0.22 / 11.
    • Source: Bedrock agreement offer rate card offer-agsprgw6z3dg6, read 2026-11-11. The cache read is 0.025x the base input price, as for Fable 5.1.
  • Fable 5.2 limits are a 1,000,000 context window and 128,000 max output. Effort values are low, medium, high, xhigh and max, with no none.
  • GPT-6.2 prices come from each model's Bedrock agreement offer rate card: Sol offer-7vsvjk6xmsl7c, Astra offer-i5wczxyjg7weu.
  • The GPT-6.2 entries list input text and image only. The 6.1 entries also list pdf, but both 6.2 models return 400 on PDF input.
  • Specs:
    • New spec/ruby_llm/models_fable_5_2_spec.rb checks prices, efforts and limits for the three ids. It also checks that the claude-fable-5-2 alias resolves to anthropic and that the us. id resolves on bedrock with an input price of 8.8.
    • models_bedrock_regional_pricing_spec.rb adds the us. id and a note that the 0.025x cache read also holds for Fable 5.2.
    • models_gpt_6_bedrock_spec.rb adds both 6.2 ids and aliases, plus an example that their input is text and image only. Its header notes the context window: 904,217 tokens accepted, about 935,000 rejected.
    • mantle_responses_spec.rb and bedrock_spec.rb add both 6.2 ids to their routing id lists.
  • The invoke server-features table lists only Opus 5.5 and Sonnet 5.5, so Fable 5.2 was deliberately left out of it.

Commit f2fded9: mantle spec and cassettes

  • The first recording attempt failed with "The security token included in the request is invalid", so commit 1 shipped without cassettes. This was not a credentials denial. spec/support/rubyllm_configuration.rb:17-20 reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN from the environment and falls back to 'test', and those were unset. The cassettes were then recorded through /tmp/with_creds.rb, which reads the ECS container credentials endpoint and execs the command.
  • spec/ruby_llm/protocols/responses_gpt6_mantle_spec.rb gets two new gpt6_models rows for 6.2 Sol and 6.2 Astra. It also gets a table-driven "rejects effort none on " example over 6.1 Sol, 6.2 Sol and 6.2 Astra. The expected errors are:
    • 6.1 Sol: 'none' is not supported
    • 6.2 Sol: does not support reasoning effort 'none'
    • 6.2 Astra: Supported values are: 'low', 'medium', 'high', 'xhigh', and 'max'
  • 14 cassettes were added under spec/fixtures/vcr_cassettes/: six per 6.2 model, plus one effort-none rejection per model. The seventh per-model example (the tool loop) is disabled, as explained below, so there are not seven per model as in PR GptTerra W1-01: Register GPT-6.1 Terra in the fork's Bedrock catalog #39.

Things a reviewer should know

  • Tool loop is turned off for both 6.2 models (tool_loop: false).
    • In the recording run, 14 of 16 examples passed. Both tool-loop examples failed at the reasoning-replay check.
    • The spec comment says the tool-loop replies of 6.1 Sol, 6.2 Sol and 6.2 Astra carried no reasoning item when recorded. So replays every reasoning item through a multi-round tool loop is gated by if traits[:tool_loop] and does not run for them.
    • Both failed at expect_reasoning_replayed (responses_gpt6_mantle_spec.rb:69) with expected 0.positive? to be truthy, got false: no reply in the loop carried a reasoning item. 6.1 Sol already had tool_loop: false for the same reason.
    • sol: true is kept for both 6.2 models. That trait only affects the commentary check inside the gated tool-loop example.
  • Both 6.2 models behave like GPT-6.1 Sol, not Terra. Effort none and minimal return 400. The lowest valid effort is low.
  • Mantle availability. Both models returned 200 with no Bedrock agreement on file. Availability reports the agreement and entitlement as NOT_AVAILABLE, the same as earlier GPT models on mantle.
  • Cassette safety. grep -lE 'ASIA[A-Z0-9]{12}|AKIA[A-Z0-9]{12}' spec/fixtures/vcr_cassettes/*6_2* printed nothing. Session tokens are filtered from the cassettes.
  • Cross-model reuse. The compaction example uses (gpt6_models.keys - [model]).first as the "other" model, which is openai.gpt-6-luna. So the 6.2 cassettes include a call to Luna.
  • Fable 5.2 live behavior (probed by hand, not covered by cassettes).
    • It thinks by default, with the thinking text omitted.
    • thinking.type of disabled or enabled returns 400.
    • temperature and top_p return 400.
    • tool_choice of tool or any returns 400. auto works.
    • max_tokens of 128,001 is rejected, and so is a prompt over 1M tokens.
    • It takes text and image input and is inference-profile only. It has been ACTIVE since 2026-10-14. The us. profile is active in us-east-1, us-east-2 and us-west-2.
    • The Fable 5.2 specs only check catalog data. No cassettes were recorded for Fable 5.2 on the Anthropic or Bedrock paths.

Verification

  • Offline run: the 7 changed or related spec files gave 102 examples and 0 failures on seeds 1, 42 and 9001.
  • bedrock_spec.rb gave 43 examples and 0 failures on the same seeds after rubocop -a. Rubocop was clean.
  • Recording: ruby /tmp/with_creds.rb bundle exec rspec spec/ruby_llm/protocols/responses_gpt6_mantle_spec.rb -e 'openai.gpt-6.2' with LANG=C.UTF-8 LC_ALL=C.UTF-8 and CI unset. That gave 14 of 16 passing. The two failures were the tool-loop examples, which are now gated off.
  • Replay after the spec change, with CI=1: responses_gpt6_mantle_spec.rb gave 42 examples and 0 failures on seeds 1, 42, 9001 and 7.
  • Only targeted specs were run, not the full suite, as the reviewer asked.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the Contributing Guide
  • This aligns with RubyLLM's focus on LLM communication
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my specific use case

Required for new features

No linked issue was found. This work is tracked internally as "NewModels W1-01".

  • I opened an issue before writing code and received maintainer approval
  • Linked issue: #___

Quality check

  • I ran overcommit --install and all hooks pass
  • I tested my changes thoroughly
    • For provider changes: Re-recorded VCR cassettes with bundle exec rake vcr:record[provider_name]
    • All tests pass: bundle exec rspec
  • I updated documentation if needed
  • I didn't modify auto-generated files manually (models.json, aliases.json)

AI-generated code

  • I used AI tools to help write this code
  • I have reviewed and understand all generated code (required if above is checked)

API changes

This is a registry and data change plus tests. It adds no new public methods or classes and changes no method signatures.

  • Breaking change
  • New public methods/classes
  • Changed method signatures
  • No API changes

Adds five registry entries (claude-fable-5-2, global./us.anthropic.claude-fable-5-2,
openai.gpt-6.2-sol, openai.gpt-6.2-astra) and three aliases, so Supernova can price
these models and read their efforts and limits. The GPT-6.2 entries list text and image
input only, since mantle rejects PDF input for both.

Co-authored-by: Sam Boland <sam.boland@appfolio.com>
@samuelboland

Copy link
Copy Markdown

Review of 5a39210. The registry change is correct: I re-ran the insertion check against aa95f613. Five entries were added, they match the brief exactly, and no existing entry changed. One part of the brief is still missing.

  1. spec/ruby_llm/protocols/responses_gpt6_mantle_spec.rb is unchanged, and no cassettes were recorded. The live call failed with "The security token included in the request is invalid." That error does not mean credentials were denied. spec/support/rubyllm_configuration.rb:17-20 reads AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN from the environment and falls back to 'test'. Those variables are not set in the container, so the request was signed with test. The container's role credentials are served by the ECS container credentials endpoint. The GPT-6.1 Terra PR (GptTerra W1-01: Register GPT-6.1 Terra in the fork's Bedrock catalog #39) recorded its seven cassettes in this same kind of container with the wrapper below, which reads them and execs the command:

    # /tmp/with_creds.rb
    require 'net/http'
    require 'json'
    require 'uri'
    full = ENV['AWS_CONTAINER_CREDENTIALS_FULL_URI'] ||
           (ENV['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI'] && "http://169.254.170.2#{ENV['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI']}")
    abort 'no container credentials endpoint' unless full
    uri = URI(full)
    req = Net::HTTP::Get.new(uri)
    tok = ENV['AWS_CONTAINER_AUTHORIZATION_TOKEN'] ||
          (ENV['AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE'] && File.read(ENV['AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE']).strip)
    req['Authorization'] = tok if tok
    j = JSON.parse(Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') { |h| h.request(req) }.body)
    ENV['AWS_ACCESS_KEY_ID'] = j['AccessKeyId']
    ENV['AWS_SECRET_ACCESS_KEY'] = j['SecretAccessKey']
    ENV['AWS_SESSION_TOKEN'] = j['Token']
    exec(*ARGV)

    From the repository root, with export LANG=C.UTF-8 LC_ALL=C.UTF-8, run it as ruby /tmp/with_creds.rb bundle exec rspec spec/ruby_llm/protocols/responses_gpt6_mantle_spec.rb -e 'openai.gpt-6.2'. Do not set CI, because record: :once needs it unset. Then:

    • Re-apply the edits that were reverted: the two gpt6_models rows (lowest_effort: 'low', sol: true, tool_loop: true), and the table-driven "rejects effort none" example over 6.1 Sol, 6.2 Sol and 6.2 Astra (the version you wrote before the revert was right).
    • Record the cassettes for both 6.2 ids. The 6.1 Terra PR added seven per model.
    • Apply the brief's rules on sol / tool_loop: if a model gives no commentary text or the tool loop fails because of the model's behavior, set that trait to false and say why in the PR.
    • Before pushing, run grep -lE 'ASIA[A-Z0-9]{12}|AKIA[A-Z0-9]{12}' spec/fixtures/vcr_cassettes/*6_2*. It must print nothing (VCR's filter_sensitive_data replaces the keys once they are in the environment).
    • Re-run the file with CI=1 on three seeds so the cassettes replay.
  2. PR description.

    • "The source of the GPT-6.2 prices was not found" is wrong. They come from each model's Bedrock agreement offer rate card: GPT-6.2 Sol offer-7vsvjk6xmsl7c, GPT-6.2 Astra offer-i5wczxyjg7weu (the brief's Prices table). Say that.
    • Replace "the exact commands and counts are not recorded" with the runs you did: the 7 changed or related spec files gave 102 examples, 0 failures on seeds 1, 42 and 9001; bedrock_spec.rb gave 43 examples, 0 failures on the same seeds after rubocop -a. Add the mantle spec counts once item 1 is done.
    • Remove "the live probes above are the evidence" once the cassettes exist.

Verified and fine:

  • The models.json insertion (count 1243 -> 1248; the new entries match the brief field for field).
  • The three aliases.
  • models_fable_5_2_spec.rb.
  • The regional pricing row and footnote.
  • The models_gpt_6_bedrock_spec.rb table, alias and no-pdf examples.
  • The id lists in mantle_responses_spec.rb and bedrock_spec.rb.
  • Leaving Fable 5.2 out of the invoke server-features table (5.1 is not in it either).

@staging-supernova-dx-appf-io

Copy link
Copy Markdown
Author

@samuelboland

Review of 5a39210. The registry change is correct: I re-ran the insertion check against aa95f613. Five entries were added, they match the brief exactly, and no existing entry changed. One part of the brief is still missing.

  1. spec/ruby_llm/protocols/responses_gpt6_mantle_spec.rb is unchanged, and no cassettes were recorded. The live call failed with "The security token included in the request is invalid." That error does not mean credentials were denied. spec/support/rubyllm_configuration.rb:17-20 reads AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN from the environment and falls back to 'test'. Those variables are not set in the container, so the request was signed with test. The container's role credentials are served by the ECS container credentials endpoint. The GPT-6.1 Terra PR (GptTerra W1-01: Register GPT-6.1 Terra in the fork's Bedrock catalog #39) recorded its seven cassettes in this same kind of container with the wrapper below, which reads them and execs the command:

    # /tmp/with_creds.rb
    require 'net/http'
    require 'json'
    require 'uri'
    full = ENV['AWS_CONTAINER_CREDENTIALS_FULL_URI'] ||
           (ENV['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI'] && "http://169.254.170.2#{ENV['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI']}")
    abort 'no container credentials endpoint' unless full
    uri = URI(full)
    req = Net::HTTP::Get.new(uri)
    tok = ENV['AWS_CONTAINER_AUTHORIZATION_TOKEN'] ||
          (ENV['AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE'] && File.read(ENV['AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE']).strip)
    req['Authorization'] = tok if tok
    j = JSON.parse(Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') { |h| h.request(req) }.body)
    ENV['AWS_ACCESS_KEY_ID'] = j['AccessKeyId']
    ENV['AWS_SECRET_ACCESS_KEY'] = j['SecretAccessKey']
    ENV['AWS_SESSION_TOKEN'] = j['Token']
    exec(*ARGV)

    From the repository root, with export LANG=C.UTF-8 LC_ALL=C.UTF-8, run it as ruby /tmp/with_creds.rb bundle exec rspec spec/ruby_llm/protocols/responses_gpt6_mantle_spec.rb -e 'openai.gpt-6.2'. Do not set CI, because record: :once needs it unset. Then:

    • Re-apply the edits that were reverted: the two gpt6_models rows (lowest_effort: 'low', sol: true, tool_loop: true), and the table-driven "rejects effort none" example over 6.1 Sol, 6.2 Sol and 6.2 Astra (the version you wrote before the revert was right).
    • Record the cassettes for both 6.2 ids. The 6.1 Terra PR added seven per model.
    • Apply the brief's rules on sol / tool_loop: if a model gives no commentary text or the tool loop fails because of the model's behavior, set that trait to false and say why in the PR.
    • Before pushing, run grep -lE 'ASIA[A-Z0-9]{12}|AKIA[A-Z0-9]{12}' spec/fixtures/vcr_cassettes/*6_2*. It must print nothing (VCR's filter_sensitive_data replaces the keys once they are in the environment).
    • Re-run the file with CI=1 on three seeds so the cassettes replay.
  2. PR description.

    • "The source of the GPT-6.2 prices was not found" is wrong. They come from each model's Bedrock agreement offer rate card: GPT-6.2 Sol offer-7vsvjk6xmsl7c, GPT-6.2 Astra offer-i5wczxyjg7weu (the brief's Prices table). Say that.
    • Replace "the exact commands and counts are not recorded" with the runs you did: the 7 changed or related spec files gave 102 examples, 0 failures on seeds 1, 42 and 9001; bedrock_spec.rb gave 43 examples, 0 failures on the same seeds after rubocop -a. Add the mantle spec counts once item 1 is done.
    • Remove "the live probes above are the evidence" once the cassettes exist.

Verified and fine:

  • The models.json insertion (count 1243 -> 1248; the new entries match the brief field for field).
  • The three aliases.
  • models_fable_5_2_spec.rb.
  • The regional pricing row and footnote.
  • The models_gpt_6_bedrock_spec.rb table, alias and no-pdf examples.
  • The id lists in mantle_responses_spec.rb and bedrock_spec.rb.
  • Leaving Fable 5.2 out of the invoke server-features table (5.1 is not in it either).

I've queued the remaining work (entry 20447) and the coder is on it. It covers these steps:

  • Spec edits: Re-apply the two gpt6_models rows and the table-driven "rejects effort none" example in responses_gpt6_mantle_spec.rb.
  • Recording: Record the 6.2 cassettes with your /tmp/with_creds.rb wrapper and CI unset. It will set sol or tool_loop to false for any model that fails on its own behavior, and say why in the PR.
  • Checks before pushing: Run the ASIA/AKIA grep, which must print nothing. Then replay the mantle spec with CI=1 on three seeds.
  • PR description: Name the offer-7vsvjk6xmsl7c and offer-i5wczxyjg7weu rate cards as the price source. Replace the "commands and counts not recorded" line with the 102-example and 43-example runs, plus the mantle counts. Drop the "live probes are the evidence" line once the cassettes exist.

I haven't run anything yet, so none of this is confirmed.


🔍 View trace

Adds both 6.2 ids to the GPT-6 mantle table and checks effort none against 6.1 Sol,
6.2 Sol and 6.2 Astra. Both 6.2 models return no reasoning items in the tool loop, as
6.1 Sol does, so tool_loop is false for them.
@staging-supernova-dx-appf-io

Copy link
Copy Markdown
Author

@samuelboland

Review of 5a39210. The registry change is correct: I re-ran the insertion check against aa95f613. Five entries were added, they match the brief exactly, and no existing entry changed. One part of the brief is still missing.

  1. spec/ruby_llm/protocols/responses_gpt6_mantle_spec.rb is unchanged, and no cassettes were recorded. The live call failed with "The security token included in the request is invalid." That error does not mean credentials were denied. spec/support/rubyllm_configuration.rb:17-20 reads AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN from the environment and falls back to 'test'. Those variables are not set in the container, so the request was signed with test. The container's role credentials are served by the ECS container credentials endpoint. The GPT-6.1 Terra PR (GptTerra W1-01: Register GPT-6.1 Terra in the fork's Bedrock catalog #39) recorded its seven cassettes in this same kind of container with the wrapper below, which reads them and execs the command:

    # /tmp/with_creds.rb...

Pushed f2fded9.

Changed 15 files (+3049 −6): spec/fixtures/vcr_cassettes/protocols_responses_openai_gpt-6_2-astra_compacts_a_short_conversation_through_compact_and_continues_from_it.yml, spec/fixtures/vcr_cassettes/protocols_responses_openai_gpt-6_2-astra_compacts_on_the_server_when_the_input_passes_the_threshold_and_replays_the_compaction.yml, spec/fixtures/vcr_cassettes/protocols_responses_openai_gpt-6_2-astra_completes_at_the_lowest_and_highest_effort.yml, spec/fixtures/vcr_cassettes/protocols_responses_openai_gpt-6_2-astra_rejects_an_over-length_input_and_accepts_a_very_high_max_output_tokens.yml, spec/fixtures/vcr_cassettes/protocols_responses_openai_gpt-6_2-astra_reports_cache_writes_and_cache_reads_separately_from_input.yml, spec/fixtures/vcr_cassettes/protocols_responses_openai_gpt-6_2-astra_streams_a_server_compaction_into_the_same_content_a_sync_reply_holds.yml, spec/fixtures/vcr_cassettes/protocols_responses_openai_gpt-6_2-sol_compacts_a_short_conversation_through_compact_and_continues_from_it.yml, spec/fixtures/vcr_cassettes/protocols_responses_openai_gpt-6_2-sol_compacts_on_the_server_when_the_input_passes_the_threshold_and_replays_the_compaction.yml, spec/fixtures/vcr_cassettes/protocols_responses_openai_gpt-6_2-sol_completes_at_the_lowest_and_highest_effort.yml, spec/fixtures/vcr_cassettes/protocols_responses_openai_gpt-6_2-sol_rejects_an_over-length_input_and_accepts_a_very_high_max_output_tokens.yml and 5 more


— via Coder session (view)

@samuelboland

Copy link
Copy Markdown

Review of f2fded9. The code and cassettes are correct. Three statements in the PR description are wrong. Please correct them in your reply so the description regenerates:

  1. "16 cassettes were added". The commit adds 14: six per 6.2 model plus the two effort-none rejections. The same bullet's own arithmetic (12 + 2) gives 14.
  2. "I did not see the full failure output, so I can't say exactly what the models returned." The recording log shows both tool-loop failures at expect_reasoning_replayed (responses_gpt6_mantle_spec.rb:69) with expected 0.positive? to be truthy, got false: no reply in the loop carried a reasoning item. State that instead.
  3. "The post-fix replay counts for the mantle spec are not recorded here, and neither is whether CI=1 was set." Your run recorded them: with CI=1, responses_gpt6_mantle_spec.rb replays at 42 examples, 0 failures on seeds 1, 42 and 9001 (and seed 7). State that.

Verified and fine:

  • Spec delta since 5a39210: the two gpt6_models rows (lowest_effort: 'low', sol: true, tool_loop: false), the comment, and the table-driven effort-none example. The 6.1 Sol example keeps its name, so its existing cassette still matches.
  • The 14 cassettes at f2fded9: no AKIA/ASIA key matches. All 30 Credential= values read <AWS_ACCESS_KEY_ID> and all 30 session tokens are filtered. The rejection cassettes carry the error text each matcher expects.
  • tool_loop: false is backed by the recorded failure, the same as 6.1 Sol.
  • The cross-model Luna call claim holds: gpt-6-luna appears in the 6.2 cassettes.
  • Registry and offline specs are unchanged from round 1 (already verified).

@staging-supernova-dx-appf-io

Copy link
Copy Markdown
Author

@samuelboland

Review of f2fded9. The code and cassettes are correct. Three statements in the PR description are wrong. Please correct them in your reply so the description regenerates:

  1. "16 cassettes were added". The commit adds 14: six per 6.2 model plus the two effort-none rejections. The same bullet's own arithmetic (12 + 2) gives 14.
  2. "I did not see the full failure output, so I can't say exactly what the models returned." The recording log shows both tool-loop failures at expect_reasoning_replayed (responses_gpt6_mantle_spec.rb:69) with expected 0.positive? to be truthy, got false: no reply in the loop carried a reasoning item. State that instead.
  3. "The post-fix replay counts for the mantle spec are not recorded here, and neither is whether CI=1 was set." Your run recorded them: with CI=1, responses_gpt6_mantle_spec.rb replays at 42 examples, 0 failures on seeds 1, 42 and 9001 (and seed 7). State that.

Verified and fine:

  • Spec delta since 5a39210: the two gpt6_models rows (lowest_effort: 'low', sol: true, tool_loop: false), the comment, and the table-driven effort-none example. The 6.1 Sol example keeps its name, so its existing cassette still matches.
  • The 14 cassettes at f2fded9: no AKIA/ASIA key matches. All 30 Credential= values read <AWS_ACCESS_KEY_ID> and all 30 session tokens are filtered. The rejection cassettes carry the error text each matcher expects.
  • tool_loop: false is backed by the recorded failure, the same as 6.1 Sol.
  • The cross-model Luna call claim holds: gpt-6-luna appears in the 6.2 cassettes.
  • Registry and offline specs are unchanged from round 1 (already verified).

Here are the three corrections, which replace the matching statements in the description:

  1. Cassette count: The commit adds 14 cassettes, not 16. That is six per 6.2 model (12) plus one effort-none rejection per model (2). The tool-loop example is disabled, so there is no seventh per-model cassette as in PR GptTerra W1-01: Register GPT-6.1 Terra in the fork's Bedrock catalog #39.

  2. Tool-loop failure: In the recording run, both 6.2 tool-loop examples failed at expect_reasoning_replayed (responses_gpt6_mantle_spec.rb:69) with expected 0.positive? to be truthy, got false. No reply in the loop carried a reasoning item. That is why tool_loop: false is set for both models, as it already is for 6.1 Sol.

  3. Replay counts: With CI=1 set, responses_gpt6_mantle_spec.rb replays at 42 examples, 0 failures, on seeds 1, 42, 9001 and 7. The "replay counts are not recorded" line should be replaced with this.

@samuelboland
samuelboland marked this pull request as ready for review November 18, 2026 22:18
@samuelboland
samuelboland requested a review from a team as a code owner November 18, 2026 22:18

@samuelboland samuelboland left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at f2fded9: registry insertion only (1243 -> 1248), 14 cassettes with credentials filtered, mantle spec traits and effort-none table, CI green.

@samuelboland
samuelboland merged commit f95ce76 into main Nov 18, 2026
24 of 25 checks passed
@staging-supernova-dx-appf-io

Copy link
Copy Markdown
Author

🤖 Supernova Code Review — View trace

🔄 Re-run this review

— via automation "Code Review Opus 5.5 Medium" (run), installed by Sam Boland (sam.boland@appfolio.com)

@staging-supernova-dx-appf-io staging-supernova-dx-appf-io Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed PR #46 at f2fded9 and found nothing to fix. No inline comments.

What I checked:

  • models.json entries: the three Fable 5.2 entries follow the Fable 5.1 pattern. The anthropic entry lists text, image and pdf input plus citations. The bedrock entries list text and image only, and the us. entry adds streaming. Prices are internally consistent:
    • The cache read is 0.025x input on all three ids (8 → 0.20, 8.8 → 0.22).
    • The cache write is 1.25x input.
    • The us. rates are a 10% premium over the global rates.
  • GPT-6.2 prices: both use a cache read of 0.05x and a cache write of 1.25x, the same ratios as 6.1 Sol.
  • GPT-6.2 context window: the 1,050,000 window matches the probe in the spec header. Inputs of 904,217 were accepted and about 935,000 rejected, which brackets the expected 922,000 input limit (1,050,000 minus 128,000 reserved for output).
  • Aliases: claude-fable-5-2 maps to anthropic and to us. on bedrock, the same as 5.1. gpt-6.2-sol and gpt-6.2-astra map to bedrock.
  • responses_gpt6_mantle_spec.rb:
    • Every example that runs for the 6.2 models has a matching cassette, 12 in total.
    • The tool-loop example is skipped for 6.2 (tool_loop: false), so its missing cassettes don't matter.
    • The effort-none table keeps the original example name for 6.1 Sol, so its existing cassette still replays. The two new 6.2 cassettes are present.
  • Fable 5.2 specs: they look up models by raw id, so the bedrock region prefix isn't changed during lookup. The shared context with configured RubyLLM resets bedrock_region before each example, so the spec doesn't leak config into other tests.
  • Routing spec lists: the 6.2 ids added to mantle_responses_spec.rb and bedrock_spec.rb match the existing GPT-6 pattern.

🔄 Re-run this review

— via automation "Code Review Opus 5.5 Medium" (run), installed by Sam Boland (sam.boland@appfolio.com)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants